Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback

An important goal in artificial intelligence is to create agents that can both interact naturally with humans and learn from their feedback. Here we demonstrate how to use reinforcement learning from human feedback (RLHF) to improve upon simulated, embodied agents trained to a base level of competency with imitation learning. First, we collected data of humans interacting with agents in a simulated 3D world. We then asked annotators to record moments where they believed that agents either progressed toward or regressed from their human-instructed goal. Using this annotation data we leveraged a novel method - which we call "Inter-temporal Bradley-Terry" (IBT) modelling - to build a reward model that captures human judgments. Agents trained to optimise rewards delivered from IBT reward models improved with respect to all of our metrics, including subsequent human judgment during live interactions with agents. Altogether our results demonstrate how one can successfully leverage human judgments to improve agent behaviour, allowing us to use reinforcement learning in complex, embodied domains without programmatic reward functions. Videos of agent behaviour may be found at https://youtu.be/v_Z9F2_eKk4

Molecular epidemiology and expression of capsular polysaccharides in Staphylococcus aureus clinical isolates in the United States.

Staphylococcus aureus capsular polysaccharides (CP) are important virulence factors under evaluation as vaccine antigens. Clinical S. aureus isolates have the biosynthetic capability to express either CP5 or CP8 and an understanding of the relationship between CP genotype/phenotype and S. aureus epidemiology is valuable. Using whole genome sequencing, the clonal relatedness and CP genotype were evaluated for disease-associated S. aureus isolates selected from the Tigecycline Evaluation and Surveillance Trial (T.E.S.T) to represent different geographic regions in the United States (US) during 2004 and 2009-10. Thirteen prominent clonal complexes (CC) were identified, with CC5, 8, 30 and 45 representing >80% of disease isolates. CC5 and CC8 isolates were CP type 5 and, CC30 and CC45 isolates were CP type 8. Representative isolates from prevalent CC were susceptible to in vitro opsonophagocytic killing elicited by anti-CP antibodies, demonstrating that susceptibility to opsonic killing is not linked to the genetic lineage. However, as not all S. aureus isolates may express CP, isolates representing the diversity of disease isolates were assessed for CP production. While approximately 35% of isolates (primarily CC8) did not express CP in vitro, CP expression could be clearly demonstrated in vivo for 77% of a subset of these isolates (n = 20) despite the presence of mutations within the capsule operon. CP expression in vivo was also confirmed indirectly by measuring an increase in CP specific antibodies in mice infected with CP5 or CP8 isolates. Detection of antigen expression in vivo in relevant disease states is important to support the inclusion of these antigens in vaccines. Our findings confirm the validity of CP as vaccine targets and the potential of CP-based vaccines to contribute to S. aureus disease prevention

Identifying Cardiac Amyloid in Aortic Stenosis: ECV Quantification by CT in TAVR Patients.

OBJECTIVES: The purpose of this study was to validate computed tomography measured ECV (ECVCT) as part of routine evaluation for the detection of cardiac amyloid in patients with aortic stenosis (AS)-amyloid. BACKGROUND: AS-amyloid affects 1 in 7 elderly patients referred for transcatheter aortic valve replacement (TAVR). Bone scintigraphy with exclusion of a plasma cell dyscrasia can diagnose transthyretin-related cardiac amyloid noninvasively, for which novel treatments are emerging. Amyloid interstitial expansion increases the myocardial extracellular volume (ECV). METHODS: Patients with severe AS underwent bone scintigraphy (Perugini grade 0, negative; Perugini grades 1 to 3, increasingly positive) and routine TAVR evaluation CT imaging with ECVCT using 3- and 5-min post-contrast acquisitions. Twenty non-AS control patients also had ECVCT performed using the 5-min post-contrast acquisition. RESULTS: A total of 109 patients (43% male; mean age 86 ± 5 years) with severe AS and 20 control subjects were recruited. Sixteen (15%) had AS-amyloid on bone scintigraphy (grade 1, n = 5; grade 2, n = 11). ECVCT was 32 ± 3%, 34 ± 4%, and 43 ± 6% in Perugini grades 0, 1, and 2, respectively (p < 0.001 for trend) with control subjects lower than lone AS (28 ± 2%; p < 0.001). ECVCT accuracy for AS-amyloid detection versus lone AS was 0.87 (0.95 for 99mTc-3,3-diphosphono-1,2-propanodicarboxylic acid Perugini grade 2 only), outperforming conventional electrocardiogram and echocardiography parameters. One composite parameter, the voltage/mass ratio, had utility (similar AUC of 0.87 for any cardiac amyloid detection), although in one-third of patients, this could not be calculated due to bundle branch block or ventricular paced rhythm. CONCLUSIONS: ECVCT during routine CT TAVR evaluation can reliably detect AS-amyloid, and the measured ECVCT tracks the degree of infiltration. Another measure of interstitial expansion, the voltage/mass ratio, also performed well

Cosmic Physics: The High Energy Frontier

Cosmic rays have been observed up to energies $10^8$ times larger than those of the best particle accelerators. Studies of astrophysical particles (hadrons, neutrinos and photons) at their highest observed energies have implications for fundamental physics as well as astrophysics. Thus, the cosmic high energy frontier is the nexus to new particle physics. This overview discusses recent advances being made in the physics and astrophysics of cosmic rays and cosmic gamma-rays at the highest observed energies as well as the related physics and astrophysics of very high energy cosmic neutrinos. These topics touch on questions of grand unification, violation of Lorentz invariance, as well as Planck scale physics and quantum gravity.Comment: Topical Review Paper to be published in the Journal of Physics G, 50 page